Report every failed document with its ID and reason#51
Merged
stevevanhooser merged 5 commits intoMar 25, 2026
Conversation
- Stop silently skipping FileExistsError (duplicates) in dataset init; report them alongside all other failures - Add _get_doc_id() helper for robust ID extraction in error messages - Each failure now shows "- <document_id>: <reason>" (e.g. "ndi_document abc123 already exists" or DID validation errors) - Remove misleading "see preceding warnings" from downloadDataset error - Keep FileExistsError skip only in add_ingested_session (re-ingestion) https://claude.ai/code/session_015oe6ucKwCFznpu1PN1nAfz
Store add failures on dataset instance (add_doc_failures) instead of
raising from __init__, so downloadDataset() can collect them and report
every failed document ID with its specific error reason in one error.
Error output now looks like:
Downloaded 7221 documents but only 7123 were added to the dataset.
98 document(s) lost:
98 failed to add to the dataset database:
- abc123def: ndi_document abc123def already exists
- 456ghi789: <specific error from DID-python>
...
https://claude.ai/code/session_015oe6ucKwCFznpu1PN1nAfz
The 98 missing documents don't raise exceptions during database.add() -
DID-python silently fails to persist them. To diagnose:
- Cross-check uses raw DID get_doc_ids() instead of isa('base') query,
which could miss docs whose type info wasn't stored correctly
- Missing document IDs are listed in the RuntimeError message
- Full JSON of all missing documents is written to
missingDocuments.json in the dataset directory for inspection
https://claude.ai/code/session_015oe6ucKwCFznpu1PN1nAfz
Prevents test artifact databases from being tracked. https://claude.ai/code/session_015oe6ucKwCFznpu1PN1nAfz
MATLAB's jsonencode unwraps single-element cell arrays to scalars.
Documents with only one superclass (e.g. just 'base') arrive from the
cloud with superclasses as a bare dict instead of a one-element list.
DID-python's _get_superclass_str only handles lists, so meta.superclass
was stored as empty string, causing isa('base') queries to miss 98 of
7221 documents.
The fix extends _normalize_depends_on (which already handles the same
MATLAB issue for depends_on and file_info) to also normalize
document_class.superclasses.
https://claude.ai/code/session_015oe6ucKwCFznpu1PN1nAfz
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://claude.ai/code/session_015oe6ucKwCFznpu1PN1nAfz